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ABSTRACT 

The  rapid  development  of  new  sensors  and  wide-band  communications  provides  the  capability  to  collect 
enormous  amounts  of  data.  An  increasing  challenge  involves  how  to  understand  and  interpret  the  data  to 
yield  knowledge  about  evolving  situations  or  threats  (e.g.,  of  military  situations,  state  of  complex  systems, 
etc.).  New  visualization  tools  and  techniques  are  becoming  available  to  support  advanced  visualization 
including  three-dimensional,  full  immersion  display  environments  and  tools  to  support  novel 
visualizations.  Examples  include  network  system  display  tools  and  evolving  multi-sensory  situation 
environments.  Despite  the  emergence  of  such  tools,  there  has  been  limited  systematic  test  and  evaluations 
to  determine  the  efficacy  of  such  tools  for  knowledge  understanding  and  decision  making.  This  paper 
provides  an  overview  of  this  problem  and  argues  for  the  need  to  conduct  controlled  experiments.  A 
sample  experiment  is  suggested. 


1.  INTRODUCTION 

An  evolving  problem  in  intelligence  gathering  and  military  situation  and  threat  assessment  is  the  challenge 
of  accessing  and  ingesting  the  enormous  amount  of  collected  data.  The  problem  is  due  in  part  to  the  rapid 
increase  in  sensing  capability  via  national  resources,  networks  of  nano-  and  micro-scale  sensors  (e.g., 
smart  dust),  unmanned  aerial  vehicles  (UAVs),  utilization  of  individual  soldiers  as  data  collectors  and 
sensors,  and  utilization  of  web  resources  (e.g.,  open  source  information).  The  amount  of  collected  data  is 
increasing  rapidly,  with  little  regard  to  the  actual  ability  to  ingest  or  understand  the  data. 
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Transformation  from  energy  to  knowledge 


Rapid  changes  in 
information  technology 
allow  huge  collection  of 
data:  a  key  challenge 
is  transforming  data 
into  usable  knowledge 


Energy  I  [)  Signals  I  Data  I  State  vectors  I  D>  Labels  I 


Knowledge 


Figure  1 :  The  Transformation  from  Data  to  Knowledge 

Users  are  faced  with  a  situation  in  which  they  are  “drowning  in  a  sea  of  data”  but  “thirsting  for 
knowledge”.  In  addition,  multiple  analysts  may  be  working  on  parts  of  a  particular  issue  and  may  not 
have  direct  interaction  or  any  interaction  at  all.  A  traditional  view  is  that  automated  systems  must  be 
developed  to  process  all  of  the  incoming  data  and  somehow  abstract  or  represent  the  data  for  human 
understanding.  This  concept  is  illustrated  in  Figure  1. 

In  this  view,  data  collected  from  multiple  sources  are  processed  via  multiple  levels  of  data  fusion  (using 
the  Joint  Directors  of  Laboratories  model)  to  provide  results  to  be  viewed  or  analyzed  by  users. 
Processing  techniques  are  drawn  from  statistical  estimation;  pattern  recognition,  automated  reasoning,  and 
optimization  for  performing  Level- 1  through  Level-3  fusion  (see  [1]  and  [2]). 

Progress  in  data  collection  and  Level-0  and  Level- 1  fusion  processing  has  far  outstripped  the  ability  to 
support  automated  analyses.  As  a  result,  we  face  a  situation  in  which  a  huge  glut  of  data  accumulates  in 
a  data  base  to  be  somehow  analyzed  by  a  limited  number  of  analysts.  Due  to  the  structure  of  the 
intelligence  community  (IC),  law-enforcement  (in  the  areas  of  espionage  and  terrorism),  and  military 
intelligence  organizations,  collaboration  is  difficult  at  best.  In  a  number  of  cases,  analysts  do  not  know 
who  else  may  be  working  on  similar  issues  related  to  their  specific  target.  Additionally,  all  collected  data 
does  not  reside  in  a  single  repository  to  be  accessed.  The  overwhelming  glut  of  available  data  becomes 
detrimental  to  the  analysis  and  decision  process.  This  has  led  some  researchers  to  cite  the  need  for  data 
mining  techniques.  The  claim  is  that  the  application  of  one  or  more  algorithms  such  as  pattern  recognition 
techniques,  machine  learning  methods,  and  advanced  filtering  techniques  will  allow  knowledge  to  be 
gleaned  from  the  data. 

There  are  several  difficulties  with  this  suggestion. 

1.  First,  there  is  an  implicit  assumption  that  the  problem  involves  “finding  the  needle  in  the 
haystack”  or  “finding  a  nugget  of  wisdom  among  the  data”.  In  fact,  no  such  needle  or  nugget 
actually  exists.  For  example,  no  single  source  of  data  would  have  predicted  the  events  of 
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September  11,  2001.  Instead,  multiple  sources  of  data  and  information,  combined  with  human 
imagination  and  analysis  would  have  been  required  to  foresee  the  actual  events.  Knowledge 
must  be  created  and  synthesized,  rather  than  simply  “mined”. 

2.  Second,  the  process  of  creating  such  knowledge  generally  requires  multiple  humans  to 
collaborate,  each  having  different  domains  of  expertise,  rather  than  a  single  person.  Data  mining 
techniques  tend  to  support  an  individual  decision-maker  or  analyst. 


3.  Knowledge  is  inherently  a  human  product, 
rather  than  the  product  or  result  of  an 
algorithm  or  technique.  It  is  important  to 
consider  the  human  analyst  as  a  vital  part 
of  the  inference/analysis  process.  We  need 
to  determine  how  to  help  the  analyst  do  a 
better  job  at  what  they  do  best. 

4.  Finally,  the  approach  has  started  “at  the 
wrong  end”  of  the  data-to-knowledge 
chain.  The  process  of  improving  the 
analysis  process  must  start  at  the  human 
side  of  the  process  (viz.,  the  right  hand  side 
of  Figure  1)  rather  than  the  data/sensor 
side.  In  short,  we  need  to  find  ways  to 
“cross  the  longest  yard”  to  transition  from  data  inside  a  computer  (gathered  via  sensors  and 
processed  using  level-0  and  level- 1  techniques)  to  knowledge  inside  a  human  analyst’s  head. 

Thus,  instead  of  attempting  to  completely  automate  data  processing  (e.g.,  via  data  mining  techniques, 
machine  learning,  etc.)  in  order  to  understand  data,  we  suggest  that  efforts  be  made  to  focus  on 
visualization  methods  to  assist  analysts  in  creative  interpretation  and  analysis.  Indeed,  this  suggestion  has 
been  previously  made  by  multiple  authors  (M.  J.  Hall  et  al  (2000),  Blasch  and  Plano  (2002)),  resulting  in 
the  creation  of  a  new  level  of  fusion  (level  5)  for  the  Joint  Directors  of  Laboratories  (JDL)  data  fusion 
process  model.  The  new  level  introduced  into  the  JDL  model  recognizes  the  need  to  explicitly  consider 
the  role  of  the  “human  in  the  loop”  and  develop  methods  to  improve  the  link  between  the  human  part  of  a 
data  fusion  system  and  the  automated  part  of  the  system. 

Extensive  work  has  been  conducted  in  development  of  visualization  techniques.  It  is  beyond  the  scope  of 
this  paper  to  provide  a  review  of  such  methods.  However,  an  excellent  tutorial  is  available  by  K.  Andrews 
at  http://www.iicm.edu/ivis/.  A  recent  limited  survey  of  visualization  techniques  to  support  decisions  was 
conducted  by  J.  Abraham  (2006).  Recent  work  involving  3-D  environments  are  described  by  D.  Shuping 
and  W.  Wright,  and  by  Kapler  and  Wright  (2005).  An  example  of  a  full  immersion,  three-dimensional 
environment  for  displaying  environmental  data  is  shown  in  Figure  2.  This  work  was  conducted  at  the 
Pennsylvania  State  University  by  D.  Hall  et  al  (2005). 


Figure  2:  Example  of  3-D  Full  Immersion 

Environment  for  Exploring  Environmental  Data 


2.  THE  NEED  FOR  QUANTITATIVE  RESEARCH 

Over  the  past  decade,  network  visualization  tools  are  becoming  a  prominent  method  to  aid  in  the  analysis 
of  data.  Network  visualization  has  the  potential  to  become  a  powerful  tool  across  a  multitude  of  analysis 
disciplines.  However,  the  drawback  to  this  method  is  there  is  very  little  understanding  about  how  these 
tools  and  their  various  components  are  perceived  by  the  analysts.  In  addition,  it  is  not  clear  whether 
efforts  expended  to  develop  advanced  displays  and  interfaces  quantitatively  improves  the  analysis  of  data. 
Research  specific  to  network  visualization  techniques  could  aid  tool  designers  in  understanding  how  color, 
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shapes,  line  design,  organization,  and  the  cognitive  characteristics  of  the  user  will  impact  the  analysts’ 
assessment  of  the  presented  data.  Further,  these  issues  must  also  be  studied  in  relation  to  how  a  decision¬ 
maker  perceives  the  presented  data.  The  ultimate  goal  of  near-term  research  should  be  aimed  at  building  a 
comprehensive  model  to  be  used  by  network  visualization  tool  designers  in  order  to  optimize  the  tool  for 
use  by  the  human. 


An  excellent  example  of  quantitative  research  in  cognitive  aids  and  visualization  is  the  work  by  McNeese 
and  his  colleagues  (McNeese  (2003),  McNeese  and  Vidulich  (2001),  Connors  et  al  (2005)).  McNeese  has 
developed  a  systematic  approach  that  links  ethnography  studies  (direct  observations  of  analysts/users  in 
field  environments),  formal  knowledge  elicitation  to  develop  cognitive  maps  of  user  analysis  activity, 
creation  of  a  scaled  world  environment,  and  evaluation  of  prototype  cognitive  aids  and  visualization  tools 
using  human  subjects  in  a  “living  laboratory”  approach.  A  conceptual  view  of  this  approach  is  shown  in 
figure  3.  Additional  details  on  this  approach  are  available  at  http://minds.ist.psu.edu.  The  site  also 
provides  information  about  a  scaled  world  simulation  tool  called  Neo  Cities.  Neo  Cities  provides  the 
capability  to  allow  teams  to  conduct  analysis  and  decision  making  using  a  simulated  terrorist  attack  on  a 
small  city.  The  tool  allows  experimenters  to  develop  evolving  scenarios  (e.g.,  involving  increasing 
activities),  support  data  and  reports  to  the  team,  introduce  cognitive  aids  and  visualization  tools  to  support 
the  decision  making  and  analysis,  and  quantitatively  evaluate  the  results  of  such  tools  and  techniques. 
McNeese  and  his  students  have  applied  this  methodology  and  conducted  experiments  for  applications  such 
as; 


•  Emergency  Crisis  Management  Centers 

•  The  TOPOFF  Exercise,  New  York/NJ  Port  Authority 

•  Environmental  Protection  Agency:  West  Nile  Virus  Workers 

•  The  WMD/Terrorist  Readiness  Exercise:  Apollo,  PA 

•  Police  Operations,  Centre  County,  Pennsylvania. 
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Figure  4:  Concept  of  Experiments  involving  Team-Based  Agents  (Yen  (2005)) 


Other  types  of  experiments  involving  use  of  tools  to  support  situation  assessment  and  decision  making 
have  been  conducted  by  J.  Yen  and  his  colleagues  (Fan  et  al,  2005).  Yen  has  developed  an  architecture 
and  knowledge  representation  techniques  to  allow  intelligent  software  agents  to  emulate  how  human  teams 
interact.  The  concept  involves  allowing  software  agents  to  work  as  a  virtual  team,  developing  a  shared 
computational  “mental  model”  analogous  to  how  human  teams  develop  a  shared  mental  model  during 
team  decision-making.  Using  this  technique,  Yen  has  developed  cognitive  aids  to  support  military 
situation  and  threat  assessment.  His  experiments  have  involved  using  human  subjects  to  perform  decision 
making  in  simulated  environments  evaluating  whether  or  not  agent  assisted  teams  perform  better  than 
teams  not  provided  with  agent  support  (see  Figure  4). 

Thus,  these  are  examples  of  the  types  of  experiments  that  we  argue  should  be  performed  to  evaluate  the 
use  of  visualization  tools  and  techniques,  especially  for  network  visualization. 

3.  THE  NEED  FOR  A  COMPREHENSIVE  COGNITIVE  ENGINEERING 
MODEL  FOR  NETWORK  VISUALIZATION 

The  ultimate  goal  of  research  in  this  area  should  be  to  build  a  comprehensive  cognitive  engineering  model 
tailored  for  network  visualization  tools.  This  model  would  provide  a  basis  for  the  designers  of  network 
visualization  tools  in  order  to  standardize  the  use  of  color,  shapes,  and  line  design  in  order  to  build  tools 
that  can  be  quickly  learned  and  utilized  by  a  multitude  of  users.  Additionally,  these  common  visualization 
techniques  would  potentially  maximize  the  usefulness  and  efficiency  of  these  tools  while  minimizing 
potential  errors  induced  in  the  analysis  process  due  to  the  visualization  tool. 
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Network  Visualization  is  currently  being  incorporated  into  a  number  of  critical  analysis  processes  such  as 
contagious  disease  monitoring,  information  system  design,  understanding  machine  learning,  understanding 
social  networks,  etc.  The  techniques  have  the  potential  to  be  tailored  to  other  critical  disciplines  such  as 
intelligence  analysis,  environmental  monitoring,  decision  support,  and  other  areas. 

It  is  important  to  consider  the  basic  problems  with  human  reasoning  when  creating  a  model  to  address 
such  a  critically  important  pool  of  data  and  information  and  the  use  of  this  information  to  make,  in  some 
cases,  grave  decisions.  Research  into  the  area  of  human  reasoning  has  a  long  history  and  some  classic 
studies  can  point  to  critical  areas  of  concern  for  analysts.  Cognitive  psychologist  Amos  Tversky  began 
looking  at  decision  making  and  found  some  critical  and  potentially  problematic  tendencies  in  human 
reasoning.  When  humans  are  faced  with  more  alternative  than  they  feel  that  they  can  reasonably  consider, 
as  is  in  the  case  of  an  exploding  data  pool,  individuals  tend  to  take  mental  short-cuts  that  limit  rational 
decision  making  and  have  difficulty  predicting  probabilities  of  events.  A  few  biases  that  plague  the 
human  analyst  are  listed: 

1.  Humans  judge  the  probability  of  an  uncertain  event  according  to  the  representativeness 
heuristic.  Humans  judge  based  on  how  obviously  the  uncertain  event  is  similar  to  the 
population  in  which  it  was  derived  rather  than  considering  actual  base  rates  and  the  true 
likelihood  of  an  occurrence  (for  example,  airplane  crashes  being  considered  more  likely  than 
they  actually  are). 

2.  Humans  also  judge  based  on  how  easily  one  can  recall  relevant  instances  of  a  phenomenon 
and  have  particular  trouble  when  it  confirms  a  belief  about  the  self  and  the  world.  For 
example,  the  vast  media  coverage  of  airplane  accidents  and  minimal  coverage  of  automobile 
accidents  contributes  to  fear  of  flying  even  though  statistically  it  is  much  safer.  Media 
coverage  however  gives  people  dramatic  incidences  of  airplane  crashes  to  recall  more  than 
automobile  accidents. 

3.  Even  the  order  and  context  in  which  options  are  presented  to  people  influences  the  selection 
of  an  option.  People  tend  to  engage  in  risk  aversion  behavior  when  faced  with  an  option 
involving  potential  losses  and  risk  seeking  behavior  when  faced  with  potential  gains. 

4.  People  tend  to  have  an  inflated  view  of  their  actual  skills,  knowledge  and  reasoning  abilities. 
Most  individuals  see  themselves  as  “above  average”  in  many  traits  including  sense  of  humor, 
driving  abilities,  and  other  skills  even  though  not  everyone  who  reports  this  way  could  be. 

5.  Humans  have  a  tendency  to  engage  in  confirmation  biased  thinking,  in  which  analysts  tend  to 
believe  a  conclusion  if  it  seems  to  be  true  (based  on  the  context  in  which  it  is  presented)  even 
when  the  logic  is  actually  flawed. 

These  considerations  lead  to  a  general  need  for  considering  research  from  the  data  fusion  community  and 
experimental  psychology  fields  in  order  to  bridge  gaps  in  understanding  about  people  and  data. 


4.  AN  EXAMPLE  EXPERIMENT:  THE  EFFECT  OF  COLOR  IN  DISPLAYS 

As  an  example  of  types  of  experiments  that  could  be  performed,  we  note  that  traditionally  little  regard  has 
been  given  to  the  purposeful  use  of  color,  shape  or  design  in  node-link  diagrams.  Typically  these  elements 
are  chosen  for  aesthetic  purposes  as  opposed  to  aiding  in  the  facilitation  of  better  understanding.  Research 
in  the  area  of  cognitive  psychology  has  provided  insight  into  the  area  of  color  perception  of  the  individual 
that  may  have  an  impact  on  how  knowledge  is  understood  from  this  media.  Generally  speaking,  color 
perception  and  its  effect  on  mood  has  suggested  that  red  is  perceived  by  individuals  as  negative  and  tense, 
while  blue  has  been  identified  as  calm  and  cool.  These  perceptions  may  have  a  significant  impact  in  the 
behavior  of  people.  Studies  in  merchandising  and  consumer  behavior  (Bellizzi  and  Hite,  1992)  have 
supported  that  blue  shopping  environments  were  associated  with  more  purchases,  fewer  purchase 
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postponements,  and  more  browsing  than  red  shopping  environments.  Further  it  has  been  suggested  that 
this  may  be  due  to  color-mood  associations  (i.e.,  how  people  self-report  to  identify  with  color)  than  to 
actual  physiological  arousal  (i.e.,  increased  heart  rate,  increased  blood  pressure  and  stress-  hormone 
release)  in  the  individual.  'Other  studies  (Hemphill,  1996)  support  this  by  showing  that  when  individuals 
are  asked  about  color  and  reasons  for  choosing  some  colors  over  others,  that  bright  colors  were  associated 
with  positive  emotional  states  and  dark  colors  more  negative  emotional  states.  These  self-report  studies  in 
addition  to  studies  where  color  is  manipulated  suggest  that  colors  have  strong  emotional,  contextual  and 
cultural  implications.  In  this  current  project,  color  will  be  considered  in  a  western  culture  framework  in 
which  color  associations  seem  to  be  consistent. 

When  designing  tools  for  use  by  analysts,  it  is  important  to  consider  other  aspects  of  visual  perception  that 
may  affect  analysis.  In  the  instance  of  considering  color  in  the  larger  context  of  visual  selection  among 
other  factors  (i.e.,  location)  it  may  not  be  the  most  salient  factor  in  selection.  There  are  several  visual- 
attention  theories  that  would  argue  that  position,  color,  form  and  luminance  are  equivalent  operations  in 
attentional  selection.  However,  Grabbe  and  Pratt  (2004)  in  their  study  of  competing  top-down  visual 
selection  processes,  found  evidence  to  support  visual  attention  theories  that  argue  that  spatial  information 
plays  a  special  role  in  the  analysis  of  visual  arrays.  Hence  in  the  proposed  study  position  and  structure  of 
a  node-link  diagram  may  have  a  competing  impact  on  judgment  and  perception  in  addition  to  color. 

Although  it  may  be  possible  that  color  and  visual  attention  factors  could  be  manipulated  in  the  design  of 
node-link  diagrams  and  other  visual  displays  to  ensure  better  decision  making  and  analysis,  research  on 
decision  making  processes  suggest  that  biases  and  other  human  cognition  characteristics  may  be  difficult 
to  counteract.  It  would  be  easy  to  assume  that  humans  who  are  asked  to  perform  tasks  that  involve  precise 
and  insightful  decision-making  would  follow  rules  and  maximization  strategies  to  ensure  success  and 
accuracy.  However,  Schul  and  Mayo  (2003)  in  examining  binary  signal  prediction  found  that  even  when 
individuals  were  given  a  simple  decision  rule  to  maximize  success  individuals  failed  to  use  this  rule. 
Individuals  being  required  to  plan  were  more  likely  to  use  rule-based  decision-making  (and  thus  optimize 
success),  yet  having  to  justify  decisions  afterward  failed  to  increase  rule-based  decisions.  Additionally, 
Tykocinski  and  Ruffle  (2003)  showed  that  individuals  often  wait  for  non-instrumental  information  before 
making  a  sure-thing  decision.  Further  individuals  who  chose  waiting  behavior  were  overall  less  confident 
about  committing  to  a  decision. 

So  it  seems  that  people,  even  when  provided  insight  and  “sure  thing”  decisions  seem  to  have  difficulty  in 
understanding  how  to  optimize  and  decide  effectively.  Presumably  because  of  a  need  for  power  and 
control  and  wanting  to  create  elaborate  stories  that  justify  actions.  Social  psychological  study  provides  a 
broad  research  base  that  suggests  that  individuals  are  further  affected  by  the  mood  of  people  around  them 
and  even  how  they  interpret  non-descript  physiological  arousal.  These  factors  will  be  instrumental  in 
understanding  how  collaboration  of  analysts  and  how  materials  are  presented  in  a  group  setting  have  an 
effect  on  decision  making. 

4.1  Problem  Statement 

Network  visualization  is  a  rapidly  evolving  technique  employed  to  assist  in  complex  data  analysis 
problems  and  as  a  decision-making  tool.  These  new  techniques,  however,  require  an  understanding  how 
of  how  the  human  will  interpret  the  data  using  the  network  visualization.  From  the  analysts’  perspective, 
questions  regarding  the  impact  of  the  visualization  tool  on  the  analysis  process  include: 

1 .  How  does  the  use  of  color  impact  the  comprehension  of  the  presented  data? 

2.  How  does  the  organizational  pattern  of  the  network  links  and  nodes  impact  the  analysis? 

3.  How  does  the  mood  of  the  analyst  prior  to  and  after  exposure  to  a  network  visualization 
impact  interpretation  of  presented  data? 

4.  How  does  color  and  organizational  pattern  of  network  links  effect  the  analysts’ 
interpretation  of  presented  data? 
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Further  from  the  decision-makers’  perspective,  questions  regarding  the  impact  of  the  visualization  tool  on 
the  decision  process  include: 

1 .  How  does  the  use  of  color  impact  the  comprehension  of  the  presented  data? 

2.  How  does  the  organizational  pattern  of  the  network  links  and  nodes  impact  the  final 
decision? 

3.  How  does  the  mood  of  the  individuals  affect  the  decision-making  process? 

4.  How  does  the  behavior  of  the  presenter  influence  the  final  decision  and  group  process? 

5.  How  does  the  behavior  of  others  present  in  the  meeting  influence  the  final  decision? 

4.2  A  Proposed  Research  Study 

Phase  I:  Our  research  team  will  examine  the  effects  of  two  classifications  of  colors  (warm  and  cool)  and 
three  types  of  network  organizational  formats  (hierarchical,  radial,  and  random)  on  mood  and  threat 
assessment.  Research  participants  will  be  screened  for  color-blindness  and  asked  for  informed  consent  to 
participate.  Each  research  participant  will  be  measured  on  mood  before  being  presented  with  simulated 
data  in  order  to  create  a  base-line  measure,  in  order  to  rule  out  extreme  pre-existing  mood  as  a  confound. 
Upon  completion  of  this  and  a  short  demographic  questionnaire,  research  participants  will  simulate  an 
analyst  perspective  as  they  will  be  presented  with  information  at  a  computer  screen  individually  and  asked 
to  make  assessments  and  decisions  based  on  what  is  presented.  Mood  and  threat  assessment  will  be 
measured  using  a  questionnaire,  Likert  scale  format  (zero  to  seven  scale,  zero  being  “not  at  all”  and  seven 
being  ’’very  true”)  for  continuous  and  more  objective  data.  Interview  data  will  not  be  used  as  interviews 
do  not  yield  as  reliable  results.  This  data  will  be  analyzed  using  a  t-test  to  determine  significant  differences 
between  group  means.  Our  hypothesis  is  that  warm  colors  will  be  associated  with  an  increase  in  negative 
mood  states  and  increased  threat  measures,  whereas  cool  colors  will  not. 


Phase  II:  After  interpreting  and  analyzing  data  gathered  from  Phase  I,  the  decision-makers’  perspective 
will  be  investigated  using  the  same  variables  and  design,  color  and  network  organizational  format.  In 
order  to  simulate  the  decision-makers’  perspective,  research  participants  will  engage  in  a  group  meeting  in 
which  a  PowerPoint  presentation,  a  presenter  (trained  by  the  research  team  for  consistency  and  neutrality), 
and  group  discussion  in  a  neutrally  colored  and  decorated  meeting  room.  After  this  process,  research 
participants  will  be  measured  on  mood  and  threat  assessment  as  in  Phase  I.  It  is  our  intention  to  use  a 
different  group  of  research  participants  in  this  and  Phase  III  as  to  not  create  effects  based  on  repeated 
exposure  to  the  stimuli.  It  is  our  hypothesis  that  the  effects  of  color  and  organizational  format  will  be 
consistent  with  that  of  Phase  I  results.  It  is  our  intention  to  examine  any  differences  between  mood  and 
threat  level  for  individual  analysts  and  decision-makers. 


Phase  III:  In  the  final  phase,  our  research  team  will  investigate  the  influence  of  the  presenter  on 
interpretation  of  data.  We  will  use  the  results  from  Phases  I  and  II  to  create  a  presentation  of  node-link 
diagrams  that  are  associated  with  decreased  anxiety  and  threat  level.  In  a  small  group,  meeting-room 
environment,  research  participants  will  be  exposed  to  two  versions  of  the  same  PowerPoint  presentation, 
with  the  same  presenter  behaving  in  one  of  two  ways:  calm  and  reasonable  or  irritable  and  dramatic.  Our 
research  team  will  train  an  actor  to  play  the  two  roles  in  order  to  create  consistency  and  to  control  for 
variance.  The  use  of  deception  and  a  confederate  (a  trained  part  of  the  research  team  who  plays  various 
roles)  is  common  in  social  psychological  study  and  helps  to  isolate  social  variables.  After  assessment  of 
mood  and  threat  level,  research  participants  will  be  debriefed  appropriately.  In  this  study,  our  hypothesis 
is  that  in  the  condition  in  which  the  presenter  was  irritable  and  dramatic  that  there  will  be  an  increase  in 
negative  mood  states  and  threat  level. 
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5.  SUMMARY 

In  this  paper  we  argue  that  new  visualization  tools  and  devices  provide  the  promise  for  significant 
improvements  in  the  ability  of  analysts  to  understand  data  and  make  effective  decisions.  However, 
systematic  research  and  evaluation  will  be  required  to  determine  the  utility  of  visualization  aids  and 
cognitive  aids.  Human-based  experiments  are  needed  to  quantify  the  effects  of  factors  such  as  color,  use 
of  multiple  dimensions,  multi-sensory  effects  (e.g.,  sonification  and  haptic  interactions),  and  individual 
versus  team-based  collaboration  and  analysis.  Researchers  such  as  McNeese  and  Yen’s  groups  have 
developed  effective  approaches  for  conducting  such  research.  We  urge  the  continuation  and  further 
development  of  similar  research. 
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Presentation  Overview 
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Problem  Statement 


i>  Sensors  and  wide-band  communications  provides 
enormous  data  collection  capabilities 

0  Little  regard  for  the  actual  ability  to  ingest  or  understand 
the  data 

$  Challenge:  How  to  understand  and  interpret  the 
data  to  yield  knowJH||e  about  evolving  situations 
or  threats  B 

New  visualization  tools  available: 

0  2-dimensional  network  visualization  displays 
0  3-dimensional,  full-immersion  display  environments 
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Problem  Statement 

^Despite  the  emergence  of  advanced 
visualization  tools,  there  is  limited 
systematic  test  and  evaluation  to 
determine  the  efficacy  of  such  tools  for 

0  Knowledge  undemanding 
0  Decision  making 
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Transformation  from  Data 

Knowledge 


Transformation  from  energy  to  knowledge 
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Energy  I  D>  Signals  I  D>  Data  I  State  vectors  I  Labels 
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The  Need  for  Quantitative  Research 


0  Network  visualization  tools  are  becoming  a 
prominent  method  to  aid  in  the  analysis  of  data 

H  Powerful  potential  across  a  multitude  of  disciplines 

0  Drawback  -  Little  understanding  about  how  these 
tools  and  the  variousgcdmponents  are  perceived 
by  analysts 

0  Not  clear  whether  thesj;  efforts  expended  to 
develop  advanced  djspTdys  and  interfaces 
quantitatively  improves  the  analysis  of  data 
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The  Need  for  Quantitative  Research 

0  Research  specific  to  network  visualization 
techniques  could  aid  tool  designers  in 
understanding  how  various  tool  and  user 
characteristics  impact  the  analysts'  assessment  of 
the 

0  Color  _3| 

0  Shapes 

0  Line  design  ~ 

0  Organization 

0  Cognitive  characteristics  of  the  user 

0  These  issues  must  also  be  studied  in  relation  to 
the  decision-maker 
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The  Goal  of  Quantitative  Research 

*>  Building  a  comprehensive  model  to  be 
used  by  network  visualization  tool 
designers  in  order  to  optimize  the  tool  for 
use  by  the  human 
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Avoidance  of 
Common  Human 
Factors  Errors 
Through  ^  * 
Deliberate  Tool 
Design m 
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J.  Yen  Concept  of  Experiments 
Involving  Team-Based  Agents 
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Operation 

Effects 


Goal  of  Cognitive  Engineering  Model 

for  Network  Visualization 

©Provide  a  basis  for  designers  to 
standardize  the  use  of: 

H  Color 
HShapes 
HLine  Design 

©Standardized  tools  J^iat  can  be  easily 
learned  and  utilized  by  a  multitude  of 
users 
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Go al  of  Cognitive  Engineering  Model 

for  Network  Visualization 

@  Common  visualization  techniques  would 
potentially: 

B Maximize  usefulness  and  efficacy 

B Minimize  potential  errors  induced  in  the 
analysis  processJ^T to  the  visualization  tool 

^  •  4l 


Cristin  M.  Hall 

©  Copyrighted  2006  Sonya  A  H.  McMullen 

Tech  Reach  Inc,  Dr.  Davjd  L  HaN 


Development 


i>  Basic  problems  with  human  reasoning 
$  Cognitive  Psychologist  Amos  Tversky 

H  Studied  decision  making 

El  Identified  critical  and  potentially  problematic  tendencies 
in  human  reasoning 

5  When  humans  ar^Tciled  with  more  alternatives 
than  the  feel  they  can  ^pasonably  consider,  as  is 
the  case  of  an  exploding  data  pool,  individuals 
tend  to  take  mental  short  cuts  that: 

El  Limit  rational  decision  making 

El  Difficulties  in  predicting  probabilities  of  events 
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Biases  that  Plague  Human 

Analysts 

^Humans  judge  the  probability  of  an 
uncertain  event  according  to  the 
representativeness  heuristic 

HHow  obviously  the»uncertain  event  is  similar  to 
the  population  jj^Bpich  it  is  derived 

0  Rather  than  considering  actual  base  rates  and 
the  true  likelihood  oTan  occurrence 

t  Example:  Airplane  crashes  being  considered  more 
likely  that  they  actually  are 
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Biases  that  Plague  Human 

Analysts 

@  Humans  judgment  based  on: 

HHow  easily  one  can  recall  relevant  instances  of 
a  phenomenon 

HHave  difficulty  wl^gi£it  confirms  a  belief  about 
the  self  and  worldly 

H Example:  Vast  ana  dramatic  media  coverage 
of  airplane  accident^and  minimal  coverage  of 
automobile  accidents  contributes  to  the  fear  of 
flying  even  though  it  is  statistically  safer 
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Biases  that  Plague  Human 

Analysts 

^The  order  and  context  in  which  options 
are  presented  to  people  influences  the 
selection  of  an  option 

0  People  tend  to  engage  in  risk  aversion 
behavior  when  faced^with  an  option  involving 
potential  losses^^l 

0 People  tend  towards-risk  taking  behavior  when 
faced  with  potential  gains 

@  People  tend  to  have  an  inflated  view  of 
their  actual  skills,  knowledge,  and 
reasoning  abilities 
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Biases  that  Plague  Human 

Analysts 

t  Humans  have  a  tendency  to  engage  in 
confirmation  biased  thinking 

HAnalysts  tend  to  believe  a  conclusion  if  it 
seems  to  be  true,  based  on  the  context  in 
which  it  is  presentp^  even  when  the  logic  is 

HAdditionally,  people^end  to  discount  data  that 
does  not  fit  into  their  hypothesis  until 
overwhelming  evidence  is  presented 
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Questions 


From  the  analysts'  perspective,  questions 
regarding  the  impact  of  the  visualization  tool  on 
the  analysis  process  include: 

H  How  does  the  use  of  color  impact  the  comprehension  of 
the  presented  data? 

El  How  does  the  organ^&tional  pattern  of  the  network 
links  and  nodes  impact  the  analysis? 

El  How  does  the  mood  of^ie  analyst  prior  to  and  after 
exposure  to  a  network  visualization  impact 
interpretation  of  presented  data? 

El  How  does  color  and  organizational  pattern  of  network 
links  effect  the  analysts'  interpretation  of  presented 
data? 
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Questions 

From  the  decision-makers'  perspective, 
questions  regarding  the  impact  of  the 
visualization  tool  on  the  analysis  process 
include: 

El  How  does  the  use  of  color  impact  the  comprehension  of 
the  presented^^^^^^H^^^^H^^^^^^^^^^H 

El  How  does  the  organisational  pattern  of  the  network 
links  and  nodes  impact  the  final  decision? 

El  How  does  the  mood  of^ne  individuals  affect  the 
decision-making  process? 

El  How  does  the  behavior  of  the  presenter  influence  the 
final  decision  and  group  process? 

El  How  does  the  behavior  of  others  present  in  the 
meeting  influence  the  final  decision? 
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Proposed  Research  Study: 

Phase  I 

Focus  -  Analysts'  Perspective 

El  Effects  of  color  -  Warm  and  cool 

El  Types  of  network  organizational  formats  -  Hierarchical, 
radial,  and  random _  _ 

Experiment: 

El  Initial  screening'^TJ^ic  demographics  data  collection, 
initial  mood  screening;  testing  for  color  blindness 

El  Research  participants  mil  simulate  an  analyst 

perspective  and  asked  to  answer  questions  based  on  a 
network  visualization  data  presented 

El  t-test  statistical  analysis 
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Proposed  Research  Study: 

Phase  II 

Focus  -  Decision-Makers  Perspective 

0  Effects  of  color  -  Warm  and  cool 

0  Types  of  network  organizational  formats  - 
Hierarchical,  radial,  and  random 

Experiment: 

0  Initial  screening  ".basic  demographics  data 
collection,  initial  m^tod  screening,  testing  for 
color  blindness 

0  Research  participants  will  engage  in  a  group 
meeting  in  which  a  PowerPoint  briefing  is 
presented  and  followed  by  group  discussion 
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Proposed  Research  Study:  Phase  III 


Focus  -  Influence  of  the  Presenter  on 
Interpretation  of  Data 

El  Using  Phase  I  and  II  data,  a  presentation  of 
node-link  diagrams  associated  with  decreased 
anxiety  and  thraat  level  will  be  developed 


Experiment: 

H  In  a  small  group,  njeeting-room  environment, 
research  participants  will  be  exposed  to  two 
versions  of  the  same  PowerPoint  presentation 
with  varying  presentation  styles  by  the  briefer 

t  Calm  and  reasonable 
9  Irritable  and  dramatic 
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Presentation  Summary 

©Statement  of  the  Problem 

^Need  for  Quantitative  Research 

^Goal  of  Cognitive  Engineering  Model  for 
Network  Visualization 

©  Considerations  mnodel  Development 

£  Proposed 
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< 


Cognitive  Engineering 
Research  ^ethodology: 

'*4-  .jr 

Proposed  Stuffy  of  Visualization 
Analysis  Techniques 
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